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ABSTRACT 


AN INSTRUMENTF.I) ARCHITF.CTURAI. SIIVIULATION SYSTF.IM 

Simulation of systems at an architectural level can offer an effective way to study critical 
design choices if (1) the performance of the simulator is adequate to examine designs execiitiim 
significant code bodies -- not just toy problems or small application fragments, (2) the details 
of the simulation include the critical details of the design, (3) the view of the design presented 
by the simulator instrumentation leads to useful insights on the problems with the design, and 
(4) there is enough flexibility in the simulation system so that the asking of unplanned 
questions is not suppressed by the weight of the mechanics involved in making changes either 
in the design or its measurement. A simulation system with these goals is described together 
with the approach to its implementation. Its application to the study of a particular class of 
multiprocessor hardware system architectures is illustrated. 
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1 INTRODUCnOISI 

SimulaUon systems are quite often developed in the context of a partiailar problem lo a 
degree, this is true for SIMPLE, an event based simulation system, and CARE, the computer 

cin^Idi The problem motivating the development of both 

SIMPLE and CARE was the performance study of 100 to 1000-element multiprocessor systems 
executing a set of signal interpretation applications implemented as "1000 rule equivalent 
expert systems" [2], 

A set of constraints pertinent to this problem governed the design of SIMPLE/CARE. The 
applications represented significant bodies of code and so simulation run times were expected 
to be an important consideration. Moreover, the issues involved with the interactions of 
multiprocessor system elements were sufficiently unexplored prior to simulation that 
simplifications in the CARE system model, specifically with respect to element interactions, 
were suspect. This need for detail was. of course, in tension with the need for simulation' 
performance. The ways ihai simulated system components would be composed into complete 
systems was initially difticult to bound. Further, it was clear that the models of these 
components would be elaborated over time and would undergo substantial change as design 
concepts evolved. It was also clear that the ways of examining the operation of these 
components would change independently (and at a great rate) as early experience indicated 
what alternative aspect of system operation should have been monitored in any given 
completed run. ^ ^ 

The design goals that emerged then were (1) that the simulation system should support the 
management ol substantial flexibilii) with regard to simulated system structure, function, and 
insirumeniation and (2) that, in order to accomplish runs in acceptable elapsed times, the detail 
of simulation should be particularly focused on the communications, process scheduling, and 
context switching support facilities of the simulated system — that is, on just those aspects of 
system execution critical to multiprocessor (as opposed to unipiocessor) operation. 


I.l Design lime Interaction And Run lime Operation 

Encapsulation of the state of design components with the procedures that manipulate that 
state is one clear way to manage design evolution. Such encapsulation partitions the design 
along well boundaries. Components (b> and large) interact with other components 

only through defined jwrts. Connections between components terminate at such ports. When 
a system simulation is initialized, connections are traced so that for every port, the simulator 
knows the connected (terminating) ports together with their containing components. Unce such 
initialization is complete, that is, throughout the simulation run, assertions about the state of a 
port of one component can be directly translated to assertions about the state of connecied 
ports of other components. 

Parntioning issues of system structure, component behavior, and instrumentation inio separate 
domains of consideration helps m managing a design that is both fluid and complex Svstem 
structure, that is, the relationship between componems. can be specified through use of an 
interactive, graphics structure editor and i.s largeK iiKlependeiil of component function per se 
Component behavior is encapsulated in a set of definitions pertinent to the given class of 
component. Each compoiieiu m a SIMPLE simulated system is a member of a class defined 
for that component type. Instrumentation i.s auiomalically and invisibly made part of the 
definition of each simulated component that is to he monitored during a run. This is done b\ 
arranging that (he class of every component to he noniiored is a specialization of the general 
instrumented^box class. The basic data structures and procedures for monitoring simulated 
components and mainiaimiig the orgamzalioiuii relationships between each component and its 
lelated instrumentation are inherited thiough this general, ancestral class and are thus made a 
separate, suhstantinlK independent consideration in the design. 


I. V S\>icni> I ,ih of Sianf(.rd Uiiueisiiv .SIMP! K 
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A further partitioning of concerns is employed to separate out the definition of the 
application programming language interface and its support (as provided by CARE) from the 
underlying information flow control governing component behavior. The behavioral 
descriptions of components (which are e.xpressed as sets of condition/action rules) deal 
generic-ally with gating information, independently of the structure of the information, betwecii 
ports of the component and its internal state variables. This is separated in the component 
model definitions from the functions performed to create and manipulate the information so 
gated. The simulated implementation of the application programming language support 
facilities, on the other hand, relies only on the specifics of the information and its structure 
and plays no part in gating it between the components of the system. Changing the definition 
of the application language is thus done independently of changing component flow control 
behavior. The application programmer and the implementer of the application language 
interface may use whatever data structures seem suitable to them, be they numbers and 
keywords or procedure bodies and execution environments. The simulation sy.stem doesn't care. 

The component probe definitions, that is, the specifications of what information should be 
captured for each component type, are separated from the descriptions of the behavior of such 
components. In designing for flexibility in the instrumentation system, it turned out to be 
important to further divide the information presentation from the information collection 
issues. The mapping from particular component probes to particular instrument panels and the 
transformations to be applied to the information as it passed from a given kind of probe to a 
given panel (and between panels) is captured in the instrument specification. This is a 

definition of what kinds of panels are included in an instrument, how they fit on an 
instrument screen, how they are labeled and scaled, and what information from which kinds of 
probes are displayed on each panel. The instrument specification also indicates what kinds of 
probes are to be connected to which kinds (that is, which classes) of components in the system. 



•ppllcatl^ cods 


programming language 
interface 
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inatrument 
"a p e c I f I c a 1 1 o n 
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Figure I: Design Time I.'Ueractioi's and Run Time Representations 


Putting together all the definitions of components, component probes, panels, instruments, 
applications interfaces, and iiuer-componeiu relationships is done in a set of design time 
interactions by a system architect, The.se interactions are used by the simulation system to 
generate efficient run time representations so that simulation performance goals can be met. 
Figure 1 illustrates the partition between design time interactions and simulation run time 
operation. Structure editing pulls together components from the component library to produce 
a circuit. Associated with some components in the library, there are definitions for the syntax 
and underlying mechanisms of a multiprocessor applications language. These specify the 
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interface used to provide the program input to the multiprocessor system being simulated.’ 
The definitions used to generate component probes are associated with each library component 
to be monitored. There may be several such definitions, each appropriate to measuring a 
different aspect of the associated component's operation. An instrument specification selects 
from these definitions, elaborates them with selections from a set of probe operation modules 
to include any pre-processing (for example, a moving average) to be calculated In the probe, 
and indicates under what conditions what information from the probe is to be sent to which 
panels of the instrument and how it is to be transformed and displayed there. Instrument 
specifications also partition the screen among the panels of the instrument. The end product 
of these design time interactions is an instrumented circuit and an instrument. The instrument 
comprises a set of instrument panels and a set of constraints relating them to the instrument 
screen. The instrumented circuit ties together instances of components, probes, and panels for 
a simulation run. 

For each defined class of component and its associated probes, the design time interactions 
produce code bodies that accomplish simulation operations during a run. It is an attribute of 
the underlying Lisp base of the simulation system that changes in these definitions ha\e 
immediate effect even during a simulation run -- an important capability during debugging. 


2 STRUCrURR AIND COIMPOSn ION 

Design time interactions to specify a syslem include the establishment of component 
relationships. Such specifications can be said to accomplish the composition of the .system 
from its components and so define its structure. SIMPLE supports hierarchical composition: 
components may be described in terms of a fixed set of relationships among their sub- 
components. Additionally, such composite components may have function beyond what can be 
inferred strictly from their composition. All this can then be included a higher level 
composite (as shown in figure 2) and so on indefinitely until the top level "circuit", the system 
structure, is reached. 
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Figure 2; Hierarchical Composition 


The behavior induced on a composite component from its parts changes according to the 
behavior of its parts. Thus, for example in figure 2, if at any time during a simulation the 
function of CAKE operator components is changed by redefining their operation, the behavior 


"The l.nuju.iiic pllllnluc^ lUl'plied i.iii I'e ii'Cil In ilehne iiuillipi<n.eiMir hiiiguaiie Milerfiu.e> Ini eilhei ■>luiieil-siiriiil)le 
nf value- p.iysiiie p.irailii.'iii') 'iipplieil. Ihe laii^ui.iiie nuerl Ke Piiill un (heve pniiiiUve^ iiippuris value-pavsiiig oil 
viicaiiis heiwteii nbicvo I'lU .luerii.iiue lmel^lve^ van lie Mud have hceii) eajilv defined in lernb nl the eiveii 
prinmoei 
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of the nine-site grid is in immediate correspondence.-’ 

Composition is described graphicallv and inleractivels in SIMPLE by picking a previously 
specified componeiu type from a menu, placing it in relationship to other components with 
"mouse" movement.s. and, through the same means, specifying the connections between its 
selected ports and those of other components (as indicated in figure 3). 



Figure 3: Graphic Structure Specification 


Through another menu selection, ports can be defined for the new composite componeiu so 
that it, in turn, can be fitted into yet higher level structures. Such external ports can be 
connected directly to ports of sub-components "within" the composite. If this is done, 
information appearing on that external port will be the responsibility of the connected sub- 
component. By this same means, a component previously described as a base level componeiu, 
can be redefined as a composite of yet lower level elements as its design is elaborated with 
further details. 

Components and (internal) connections can also be deleted from a library componeiu and 
replaced with substitute components. After all sub-components and connections have been 
added, deleted, elaborated, and replaced as required, the completed structure can then be entered 
into a library of components and used in turn to compose higher or equivalent level 
components. 

2.1 CARP. Rase Components 

CARE supplies a small library of system level base componeiu types. Currently these are the 
nst-input, the net-output, the fifo- buffer, the operator, and the evaluator. The net-input, net- 

■’However. tor fciisoiu ujiK'erning siimiloiioii perl'oriiuiik'c .iiul betuuic of iheir relalively low f requeue), vhiingeb in 
ihe miiiibcr :ind iiiimei of the miernnl Viiruiblcb uf ujmponcnb iind ihc siriicuiral rdaliun^hips btrlweeii sub- 
compuntfiUs of a conipusiltf art iiul rcflcucd m an alrcaO> iiisUiiuiulcd ciMiil. Changes iii the miernal siruclure ol a 
CARF lihrars aniiptnteiU. for exanipte. ^^ill ht: reflcUeO onl> in circuUs iiisianUiUcd after ihc change look cftecl. 
For ihiD reason and lo reduce lung term storage requiretneiUs and load time fur the fuiidaniciUaII\ iterative circuits that 
we pnmarilv iUid\, we do not keep files of niAiamiaied circuits. Thc> are mslaiUiaied as needed from a high Ic^el 
lihrar> compi»nenl wuh the ^al^c proiolvpical >iriuuiie. 
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output and fifo-buffer .accept (or block), route, and buffer transmissions. They do so in 
accordance with a dyn.amic, flow-controlled, multicast, cut-through communications protocol as 
described in [3]. The evaluator does the real work of the application: evaluating the 
application of functions to their parameters. The operator does the overhetid work a.ssociated 
with such evaluations: for example, scheduling processes and sending and receiving (but not 
routing) messages. 

In keeping with the objective of focusing simulation cycles on the aspects of the simulation 
particularly relevant to multiprocessor operation, the beh.aviors of the net-input, net-ouiput, 
and fifo-buffer component classes are defined in fair detail, that is, at the register transfer 
level. Routing operations are described procedurally and assumed to occur within a time set by 
a parameter to the simulation. As indicated previously, the simulation of the operator and 
evaluator is broken into two aspects: the control of the flow of information and the functions 
performed on that information. The former is described in terms of SIMPLE behavior rules 
(as documented in section 3). regi.ster transfer by register transfer. The latter is described 
directly in terms of procedures and the simulated time taken by such procedures is modeled. 
In the case of the operator, this is done .as a function of the number of storage cells 
manipulated during an operator procedure. In the case of the evaluator, this is done as a 
function of the execution time used by the m.achine executing the simulation, that is, the 
simulation vehicle. 


2.2 CARE Composite Components 

The prototypical composite component supplied with CARE is the site. As supplied, it 
includes net-inputs and net-outputs for up :o eight "neighboring” components (generally other 
sites), a net-input .and a net-output with associated fifo-buffers for local receptions and 
transmissions, and, finally, an operator and evaluator as described above. Specializations of the 
site, for example, the torus-site, exist in the library to fit the site into alternative topologies by 
supplementing the site routing and wiring p 'v.edures .as appropriate to the topology. 


2.3 Automatic Composition in CARF. 

Although any connection of components c.an be created by the means noted previously, for 
some repetitive, well patterned systems of connections, composition can be automated. The 
CARE library includes a component, the iterated-cell. which represents a template for the 
creation of composite components by iteration of a unit cell. The unit cells (for example, the 
torus-site) are specializations of other components (for example, the site) as just discu,ssed. 
The specializations include a method for responding to a request to provide a wiring list. Such 
a list associates each source port of a cell with the corresponding destination port (in terms of 
port names) and the position of the destination cell relative to the source cell in the iterated 
structure. The iterated cell component uses this information to make the required connections 
between each of its constituent cells. 


3 SPECIFYING BEHAVIOR 

SIMPLE is an event htised simulator. The behavior of a simulated component is described in 
terms of responses to the events pertinent lo that component. A component's response may 
include consequent events to be handled by the simulator as well as direct operations on 
component state. Assertion of consequent events and the responses to them (involving further 
cons*’ ’"ces) drives the simulation. When there are no more events to handle, the simulation 
is coi i. 

To maintain modularity in a simulation system, responses to simulation events should be 
local to the affected comp*)iient and its defined ports, that is, its connection to the remainder 
of the simulated sysieiii. I'he composition system of the simulator maintains the relationship 
between ports of one component and those of other components connected to them. Assertions 
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relative to a port of a component are thus systematically translated to events pertineiii to 
components connected to it. This is the general mechanism for event propagation between 
components. In a limited number of cases, a direct operation on a related component may be 
appropriate. With fair warning about its possibility of abuse, a facility is provided to 

accomplish this. 


3.1 Behavioral Ruie.s 

The behavior of a component is described in terms of its responses to pertinent events. 
Each event stipulates the component affected, its port or state variable signalled with an 
assertion, the asserted value, and the simulated "time" of the event. The time of an event may 
be thought of as the "current" simulation time. Differences in event times represent the 
temporal relationship between events. Event times in SIMPLE simulations are monotonically 
increasing. 

For each type of component, there is a procedure to handle pertinent events. The arguments 
to the procedure are those stipulated by the event (as just described). The procedure tests for 
conditions and, as satisfied, assents or directly effects consequent actions. The conditions may 
include arbitrary predicates on the event parameters and the state variables of the component. 

Event based simulators are based on the assumption that state and port variables remain 
unchanged until explicitly modified. Synchronous designs, that is, those in which the 
opportunities for state change are temporally quantized to a clock, can be modeled in such 
implicitly asynchronous, event based simulators by asserting the clock signal on a port of each 
and every clocked component of the simulated system, If only some of i’ components in a 
system need take action on each clock signal, there is an obvious * .efficiency in this approach 
that is crippling for systems with even a modest number of components. 

If, however, event times in an event based simulator are restricted to integers, the clock can 
be assumed. All that is needed is a way to detect the event for which a boolean combination 
of conditions as strobed by an assumed clock is first met. Primitive condition predicates are 
supplied for detecting an "edge" (a value changed by the current event) with a coincident 
"level" (a value set before the current event) of two ports or state variables of a component in 
either of the two possible event sequences. The predicate both-states in the example 
evaluator behavior rule shown in figure 4 has these semantics. 

: ; If the evaluator is ready and there is at least one runnable process^ 

((or (both-states Evaluator-Status^ 'ready Evaluator-Queue-Status ’some) 
(both-states Evaluator-Status 'ready Evaluator-Queue-Status 'full)) 

; make it current, start evaluation, and adjust status as per removal. 

(setq Evaluator-Status 'busy) '.block rule 

(assert-state Evaluator-Status 'busy now) :next event 

(setq Current-Evaluation (queue-take Evaluator-Queue)) ; note process 
(user-evaluate Current-Evaluation now) ;e.xecule it 

(send self ;evaluator-queue-decreased now)) '.note change 

Figure 4; Example Condition/Action Behavior Rule 

Figure 4 iiUiMrates the generality of SIMPLE behavioral descriptions. The underlying object- 
oriented programming system. Flavors [4]. in which SIMPLE is implemented provides for 
direct reference of component state variables. The conditions and actions of behavior rules for 
a component then need only name the component’s port or state variable (as stipulated in the 
definition of that component type) to get or change the appropriate value in the component 
instance for which the event is pertinent. Actions may include arbitrary procedures: for 

example, the procedures user-evaluate and aueue-take in the given example. 
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3.2 Using Methods 

The environment for tlic cxeuition of the procedures defining responses to events include.s 
the state variables and ports of the component instance for which the event is pertinent. 
These procedures are Flavor methods [4] (in this case corresponding to the ;ApplyRules 
message) of the component type and. as just noted, refer implicitly to the state variables of the 
component instance handling the event. Other methods may be defined for simulated 
components; for example, the :evaluator-queue-decreased method invoked in figure 4. 
Such methods nave proved to be a natural way to realize the functional operations of 
components not described by behavior rules. 

The composition system leaves information about the enclosing and contained component 
instances for each simulated component in system defined state variables of that component. 
With this information, methods directly referencing the ports and state variables of such 
related components may be invoked as needed. This is a useful but sharp-edged facility. The 
warning about loss of modularity given previously applies here. 


4 IMSTRUMKINTATIOIN 

The results of a simulation are primarily the insights it provides into the operation of the 
simulated system. The "insight" we frequently experienced using an early version of the 
simulation system wa.s that more interesting results could have been produced by the run just 
completed if only the instrumentation had been different. With this in mind, the design for 
the current versiCii of the simulation instrumentation system was aimed at flexibility. This 
was attained without significant performance impact by building efficient run-time system 
structures before each run. as outlined in section 1.1, from the declarations defining the 
instrumentation. 

The organization of the instrumentation system is pictured in figure 5. The simulator 
interacts with component instances through assertions, that is, calls on an assert function, in 
behavior rules (the methods associated with :ApplyRules messages). All instrumented 
components are specializations of an insrrumenied-box (as well as other classes). After each 
invocation of lApplyRules for such components, the lApplyRules method for a generic 
instrumented-box is applied. This causes invocation of the : trigger met lod for each 
component- i)fuie associated with that component. Since this flow of measurements is 
accomplished by means invisible to the the writer of behavior methods for a component, the 
concerns surrounding coni(ionent design are effectively partitioned from component 
instrumentation. The remainder of this .section details these "invisible" means used to 
accomplish nuasurement flow during a simulation run as the measurements are staged from 
components through component probes to imstrument panels. 


4.1 Component Probes 

The first filteiing of events is done by component probes. Some events caii.se no further 
measurement activity since, as it turns out. not all events merit action on the part of the 
instrumentation system. The parameters of the event and the ports and state variables of the 
instrumented component dealing with the event are available to the component probe as are 
the state variables of the probe itself. Each piece of the selected information is tagged with an 
identifying keyword and passed along as the parameters of the : trigger method along with a 
keyword identifying the type of componen, probe, a number representing the current event 
time, and a pointer to ihe component with which llie information to he associated in the 
display. This pointer might be to .some component related to the one acf.dly handling the 
event, for example, the component enclosing it. 

Component probes may he composed of predefined probe operation modules to do standard 
calculations (for example, moving averages) and then to forward the results to selected panels. 
In order to aulomale the composition of probes to accomplish such operations, each of these 
operations is chained together by invoking the method for that probe that is as.sociated with 
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Figure 5; Instrument Svstem Organization 

the system-defined message name of the generic next operation. Thus, the : trigger method 
calls the :calculate method of the probe which, in turn, calls its :select method which, 
finally, calls the : update method of the selected panels associated with the probe. Probes are 
composed b> naming them a,s specializations of appropriate probe operation modules (for 
example a : calculate module for mo\ing averages) as desired. The default, if no 
specializations are stipulated, is to pass through information without change to all the panels 
associated with a probe. 

Information flow between components and panels is accomplished b> the component probes 
associated with each instrumented component. The creation of such componeni probes and 
their association with appropriate components (h> execution of :add methods) accomplishes 
the inslrumenlatton of a circuit. This is done v^hen an instrumen i.s created. During 
simulation initialization, the components of the circuit (and their suo-components) to he 
instrumented are (recursively) examined bv each icmplate probe defined for the instrument to 
see if they are to be monitored. If so. the :copy method for the given template probe is 
invoked to create a new instance of the appropriate component probe and add it to the probes 
connected to the component. Each template probe previous!) received the identifiers for the 
panels to which its clones should send information. These will be the panels identified when a 
component probe invokes the : update method. 



4.2 Insfrument Specificafions 

The operations performed hv an insiriimeni panel are to: 

• hind mformalion previoiish stored according to the component pointer supplied bv 
the : update method; 
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• Link new data structures as needed (to save such information) to other such 
structures of the panel: 

• Save in these data structures the results of expressions that reference indicated 
keyed information from the ; update parameters and the prior contents of the 
structures: 

• Send the results of periodic analyses on the information associated with a panel for 
display by the same panel or by some other: and 

• Show processed information in the manner specified for the panel. 

The defaults for the panel operations supply the most commonly required specifications 
implicitly, so simple operations are simply specified. These defaults can be overridden as 
needed and either predefined or user specified alternatives for the panel operations can be 
selected in their place. Arbitrarily complex (Lisp) expressions can be used to specify the 
transformations between the information provided by a probe and that saved and displayed by 
the panel. 

These transformations and all the default overrides for the panel operations that are 
stipulated in the instrument declaration are scanned when a new instrument is created for a 
simulation session. They are compiled at that time into code bodies referenced by run time 
control blocks associated with each panel. A simulated system is instrumented by examining 
all of its components and attaching to each component the copies of template probes specified 
by the instrument definition that are appropriate for the component (by means of calls on the 
icopy and ;add methods for the probe). This can be a many to many relationship as shown 
in figure 6. 


panels 

probes components 

■nrnniRfinnn 








producer-limited 

iscroliing text with scroll bar 

consumer-limited 

iscroHing text with scroll bar 



Figure (i: Instrument Probe aiul Panel Relationships 

Component probes to measure "load" and "latency" are specified in the given example for 
each operator and evaluator in the circuit. The "load" and current "connection" for etich net- 
output is also to be monitored. Some panels, for example the one showing "consumer-limited" 
processes, receive iiipuiv from only one type ol cumponeni probe, those measuring evtiluator 
latency. Others, such as the one ’iieasuring "process-latency" receive inputs from more than 
one kind of probe (in this case, from probes measuring operator latency as well ns those 
measuring evaluator latency). A way must thus he provided to distinguish the type of probe 
.sending informatioii to a ptinel: this is described iii the next section. 
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Some probes send information to onl> one panel, for example, the net-output connection 
probes. Others monitor information which is needed by several panels, for example, the 
operator latency probe, rransformation of the raw information provided by a probe will need 
to be specialized to the information expected by each panel receiving it. A general way to 
stipulate these transformations is stipulated in the next section. 


5 EXAMHLE. PANELS 

Some example panels are described in this section to give a feel for the instrumentation 
possibilities available in CARE and elaborate on how the requirenients described in the 
previous section for probe type identification at a panel and per panel specialization of the 
information provided by a probe are handled. 



5.1 Point Plot Panels 


The first panel (shown in the left half of figure 7) is an example of a point plot panol u.sed 
to generate a scatter plot. As an option, only points representing simulated activity over a 
limited past history from the most recent event time are kept for display. In this example, 
resource load^ information is provided by the operator-load and evaluator-load component 
probes attached respectively to the operators and evaluators of the system. 
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Figure 7: Point Plot and Scrolling Line Plot Panels 

The balance between the "availability" of the evaluator and operator of each site, that is. the 
complements of their respective loads, is displayed during the simulation as events are 
processed that change this measure. In order to avoid capturing information at too fine a 
temporal granularity, previously gathered information for a given site is overwritten if it is 
within a given sampling interval of the new information. Information that is beyond a given 
history range is dropped. The scale of availabilities displayed is fixed between 0 and 1.0. The 
panel specification to declare all this and to also stipulate the axis labels is shown in figure 8, 


■Rooukc loiitl Ucl Micd .!> ( I * I / ( I ♦ iiggiesiue-qiiciie-ltfiitiih)) where ihe u^iiregiile i|ueue*lenglh is ihe sum uf 
ihc Icnglhi ut* iill pru^utiii^ wurk fur the nrsuurtc. 
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’((("Operator") (0 1.0) (- 1 ( :operator-load :busy))) '.Bottom axis 
(("Evaluator") (0 1.0) ((- 1 ( :evaluator-load :dusy)))) ; Left axis 
;find (f ind-sainple-distinct (:simulator :tiine) .sampling-interval) 

:show (recent-history (:simulator :time) .point-panel-history-range 0)) 

Figure 8; Site Correlation Panel Specification 


5.2 Scrolling Line Plot Panels 

An example of a scrolling line plot panel is shown in the right half of figure 7. This panel 
sums the loads seen by the resources in the simulated system and displays this as a strip chart, 
the "system history". Some of the same probe load information used by the previous panel is 
used in this panel as well, but with different transformations defined in the panel specification 
as shown in figure 9 

’((("Simul» Time [us]") (.history-range) (:simu1ator :time)) 

(("Netwo. .") (0 .site;) (: net-output-1 oad ;busy save-sum)) 
(("Processing") (0 .sites) 

(average ( :evaluacor-load :busy save-sum) 

(: operator-load :busy save-sum))) 

:f1nd (update-history (rsimulator ;time) .sampling-interval) 

:show (recent-history (:s1mulator :time) . hictory-range 0)) 

Figure 9: System History Panel Specification 

Line plot panels may have two independently scaled vertical axes. For the system history 
panel shown, the sum of network loads as indicated by the net-output components of the 
system is plotted against the left axis and the sum of the processing loads provided by the 
current average of the sums of the operator and evaluator loads is plotted against the right 
axis. Event time is plotted on the horizontal axis. The update-history function uses the 
component pointer to find the information previously saved for that component and records 
the current event lime as the (isimulator :time) so that it may be used to display 
information correctly on the horizontal axis. The curienl sums of the evaluator load.s and the 
operator loads measured by the system are stored in a record for the given event time (or a 
prior event time within the specified sampling interval) by the calls to the save-sum function 
specified as part of the save operation. 


: Bottom 
: Left 
: Right 


5.3 Self Scaling Line Plot Panels 

Figure 10 illustrates both the self scaling of displays and the use of a display analysis 
operation. For this self scaling line plot panel, two pieces of data are collected for each 
operator in the system: the load on the operator. shovMi on the right axis, and the latency of 
(he information it has most recently received. This last item is provided by the operator 
latency probe in two parts: (1) the interval between the creation of the information and its 
receipt by the net-input fecd.ng the operator and (2) the interval between such receipt and the 
operator taking action on it. There are thus two curves plotted on the left axis. The 
specification stipulates a list for the left axis display. The elements of this list are the "net 
delay" and the sum of this measure and the "operator delay" monitored by the operator latency 
probe. Since both delays are non-negative, their sum must be at least as large as either one 
taken alone; the two curves may be superimposed but can not cross. The difference between 
the two curves is the incremental delay added by the operator. 

The panel specification for (he operator-network panel is shown in figure 11. In addition to 
transformations shown previously, an analysis function is stipulated for the send operation of 
the panel. The information saved from each of the probes sending : update messages to the 
panel is to be sorted from the greatest to the least values of the associated sum of delays 
described abme. This information is to be saved as the operate, latency rank and used as such 
to determine the position on the horizontal axis that the delay and load information will be 
displayed. 
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Fiyure 1(1: Self Scaling Line Plot Panel 

'((("Operators") (1 .sites) ('.operator-latency :rank)) 

((("Latency" "us")) (0 n11) '.Second string: 90 degree baseline shift 
((: operator-1 atency (;net-de1ay (+ :net-de1ay : operator-del ay )))) ) 
(("Load”) (0 1.0) ( :operator-1oad tbusy)) 

:send (sort-arrays 

( :operator-latency (+ :net-delay :operator-delay)))) 
( ( :operator-latency :rank)))) 

Figure II: Operator-Network Panel Specification 


5.4 Boxes and Lines Panels 

Perhaps the most intuitively satisfying of the types of panels available is the boxes and lines 
panel, a graphic representation of a circuit showing its components and their interconnections. 
An example of such a panel is shown the left part of figure 12. This class of panels uses 
information left behind by the structure editor when the circuit was defined. Its form is thus 
automatically generated. The position of the components ("boxes”) and the connection.s 
between them ("lines") in the displa> are used to animate system operation. In the example 
shown, the shading (or color) of the boxes is used to indicate the availability of the evaluators 
in the simulated system ns the simulation proceed.s. Darkest shades indicate highest availability, 
that is, empty queues for utilization of the resource; lighter shades indicate lower availability, 
that is, longer queues. The lines between boxes indicate communication paths that are in use, 
that is, not ":free" at the time of the most recent show operation for the panel. 

The panel specification for the mapping panel, an instance of a boxes and lines panel, is 
shown in figure 13. There are two specifications for the panel: one for the boxes and one for 
the lines. The specific.^tion for boxes in the panel stipulates that the availability of evaluators 
in the sites corresr/ondi'ig to the boxes displayed controls the shading of those boxes. The 
scale is defined to run ‘‘'om 0 to 1.0. The specification for lines in the panel uses the 
connection infoiination icponed for the net-output to determine line placement on the display. 
When the status is reported as ifree. the connection information is dropped from the panel 
and the corresponding lines are removed. 
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Figure 12: Boxes and Lines Panel and Scrolling Text Panel 

' ( ( ( ’’Evaluator Available") (0 1.0) (- 1 (: evaluator-load :busy)))) 
’((("Packet Trace") nil (: net-output-connect ion :points)) 

(("Packet Status") nil ( : net-output-connection :status)) 

:find (f ind-and-remove (: net-output-connect ion :status) :free))) 

Figure 13: Mapping Panel Specification 


5.5 Scrolling I’ext Panels 

Sometimes, the most appropriate wav lo displav information is to show it as text. Based on 
a similar facility provided by the underlying Lisp system, the scrolling (ext panel provides a 
scrollable window into line.s of text. In the right part of fixture 12, the delay in each process 
execution while waiting for something to do. that is. the event time interval spenl waiting for 
an appropriate task to appear on a certain stream of tasks, is shown together with the process 
that finally procliiceci the awaited work. This mformation is sorted so that the text lines 
appear from the greatest stream waiting interval to the least. 

’((() ("^4D ^A") 

((fix (: stream-waiting : interval}) \firsf field 
(let^ ((origins (packet-origin (: stream-waiting :packet))) 

(origin (if (listp origins) (first origins) origins))) 

( remote-address- local origin)))) \second field 
:send (sort-arrays ((,#’> ( : streain-wai t ing :interval))) nil)) 

Figure 14: Producer Umited Process Panel Specification 

The values aiul formats used for di.splay m a scrolling text panel are defined much as in 
previousl) defined panel.s. Formal control strings take the place of scale information, As 
usual, values are desunhed h> a list of forms, each one of which specifies the transformations 
to perform on mfurmaiion received from probes. The example specification in figure 
14 shows the gLiieralil) wiili which probe information can be incorporated in Lisp expressions 
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to produce transformation specifications. The information used to generate the value for the 
second field of the text display is based on the origin of the task packet that arrived on the 
stream the process was waiting for. 


5.6 Noting Simulation Farnmeters 

The CARE component models are parameterized through menu interaction as shown in 
figure 15 to allow easy variation of their performance characteristics relative to each other. 
Additionally, the site model parameterizes alternative routing strategies; directed, that is, 
blocking when progress can not be made toward the goal; spiraling around the goal if progress 
toward ic is blocked; and dithering, that is, routing away from the goal even if only the last 
link towards it remains to be acquired. The rate at which each site accepts application data is 
also a parameter, the data rate and can be used by an application to control ho^s hard it 
drives the simulated system. 


Sinuitatlon Parameters 


Data Rate : 

25.0 


Evaluation Override [,ws] : 

NIL 


Stack Group Sultch Override C.wS] 

: 1 .0 


Process Block Creation Override 

C^4S]: 4.0 


Stack Group Creation O^^erride Cm.s] : 2080 


operator word Touch Tine Cr^s] s 

0.2 


Connunication Cycles: 

4 


Routing; 

DIRECTED SP!RAL1N(3 DITHERIN‘3 | 

Exit □ 

Quit □ 

III } 


Fi«ure 15: Parameter Menu 

Many of the CARE parameters are specified as overrides. If not specified, the corresponding 
performance is taken as measured on the simulation machine. Thus, the evaluation override, 
that is, the time to perform an evaluation can be specified as non-nil in order to fix the time 
that each user evaluation will take. (This is useful in making runs repeatable for debugging). 
The time that it takes to .switch context can he specified as the stack group switch override. 
Similarly, the time to create a process control block and a stack context for that process can be 
taken as given rather than meiisured by specifying respectively the process block creation 
override and the stack group creation override. 

The time required for operator execution is modeled in terms of the number of words the 
operator must manipulate in handling a given message. The manipulation time per word is 
specified by the operator word touch time. La.stly. the performance of the communication 
subsystem is specified as communication cycles, I his is done in terms of the minimum 
number of evaluator data path clock times (that i.s, event times) required for a 32-bit word to 
pass a given point in the network. Thus the parametric specification, "4 communication 
cycles", dictates that 8 bits may cross such a boundary each time the evaluator pas.ses through 
one event time. If the communications path were narrower or the base communication clock 
rate were lower, a higher number would be specified. 


i/m'm MiSCM 3* OlMCtC# t, 2M8.I, »•>«, Cylluttion liM. 


Figure 16: Annotation Panel 

The last example of SIMPLE panels is the iinnoiation panel as illustrated in figure 16. This 
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is used to (automatically) record the date, time, and parameters of the simulation run as well as 
any other information the user chooses to keyboard into it. 


5,7 An lastruincnt .Screen 

All these panels are put together in an instrument screen according to a set of layout 
constraints manipulated by the underlying window system. The finished screen might look like 
figure 17. The instrument screen is redrawn at a rate set by the user. By experience, it is 
often better to update the screen at a frequency low enough to let the user interpret each 
screen comfortably than at the maximum rate possible. This approach also restricts the 
computing resources consumed by the instrumentation system. More focused approaches to 
controlling instrumentation load on the .system include the ability to freeze selected panels and 
disconnect selected probes during a simulation run. 
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Figure 17: Over.seer Instrument 


6 USING FROGRAM I)KVFXOF!VlENT lOOLS 

The SIMFLE/CARb simulation system i.s integrated into the underlying Lisp machine 
program development environment. The objects and data structures at both the component 
model and application language interface have abstraction interfaces that provide summary 
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state information when they are displayed in text form. These text abstractions are "mouse 
sensitive" in the development machine environment and so can be inspected at successively 
finer levels of detail as desired. 

In figure 18, the net-output components of the site at grid coordinates (3 2). the particulars 
of the net-output on the east side of the site (that is. net-output-3), and a summary of till 
the sub-components of the site at (3 2) are being inspected. This same kind of view into the 
progress of a simulation is provided in the debugging process and may, as shown in figure 19, 
refer to the conceptual entities of the application that is driving the simulated system. 
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Figure 18: Inspecting Simulated Components 

In the example shown in figure 19, a distributer proce.ss running on the evaluator at site 
(1 1) has made an improper call on the update-locale function during execution of its 
: start method. It might have been appropriate to investigate this situation in terms of the 
modeled components. That could be done, for example, using the debugger to inspect the 
evaluator component, its enclosing site, related net-output components, or whatever else at the 
component model level seemed relevant. In this case, what was done was to use a few mouse 
clicks to indicate interest in the source file for the distributer : start method generating 
the problem. It was brought up for review and control was then transferred to an editor using 
the underlying program development environment as shiSwn in figure 20. 

Because of the implementation system chosen for the reali 2 ation of SIMPLE/CARE, at any 
point in the simulation, procedures either in the application or in the component model.s can 
be modified, incrementally recompiled (within a few seconds), and be made effective for all 
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calls on them -- even those in the interrupted slack frame. Thus simulation execution can he 
backed up to some previous point in the .stack frame and retried (given that intermediate side 
effecting code, if any, is safely re-executabic). 
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7 CONCLUSIONS 

The goals of simulation flexibility and simulation environment completeness have been dealt 
with in the ways described throughout this paper. In summary, the system is flexible in that it 
supports; 

» Arbitrary data types and lengths in simulation. The information whose flow and 
creation is controlled by simulated components may be of arbitrary complexity 
" from numbers and keywords to procedure bodies and execution environments. 

. Instantaneous effect of definition change at both the application and component 
modeling level (even during a simulation run). 

• A broad range of instrumentation customization. Customizations may involve 
arbitrary expressions for probe data transformations, many to many probe to panel 
mappings, information from summary analyses on one panel’s data included in 
another, and control of what state is saved and for how long. 

• Separation of probe and component definitions to facilitate their independent 
modification. 

» An application language interface that is easily extended or changed without 
recasting the information flow control described by the component behaviors. 

While there is always room for additional capability^ SIMPLE/CARE is a usefully complete 
system. It now includes: 

« Supplied components for a network multiprocessor simulation with many of their 
parameters customizable by menu interactions. 

» A hierarchical structure editor that currently provides automatic grid and torus 
composition operators. (Automated composition of richer topologies, such as 
hypercubes, has been provided for in the basic design). 

• A rule language that supports a synchronous design style without incurring the 
overhead of (naive) synchronous simulation. 

• Method invocation for functional simulation that is integrated into the behavioral 
simulation rule system and which provides for operations by and on both local and 
hierarchically related component.s. 

. Method specification design aids provided by the underlying program development 
environment (for example, method dictionaries and quick access to method sources 
from the debugging system). 

. An evolved set of panel templates providing sorted, .scrollable text lines as well as 
self and fixed scaling, "two and a half" dimensioned, history sensitive displays 
which may be scatter plots, strip charts, line graphs, intensity maps, and signal 
animations. 

We set off to build a multiprocessor simulation system with performance adequate for the 
understanding of multiprocessor systems executing significant applications. The 
SIMPLE/CARE simulation system has been used to study the operation of "expert systems" of 
respectable size [2]. Depending on instrumentation load, these studies have involved 
simulation runs from 20 minutes to several hours each. While faster would surely he better, 
performance has proven adequate to these needs. 
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